Machine Learning Engineer Nanodegree

Capstone

Project- Corn Commodity Futures Price Predictor

In this project we will try to predict closing weekly price of Corn Commodity Futures. In order to perform this prediction we will create a dataset that includes weekly Corn Futures closing prices as well as Long Open Interest and Short Open Interest of Processors/Users( sometimes they are called Commercials) from COT reports and by using this dataset we will try to predict next week’s prices.

1. Data Sets

Historical Futures Prices: Corn Futures, Continuous Contract #1. Non-adjusted price based on spot-month continuous contract calculations. Raw data from CME: Can be found here
Commitment of Traders - CORN (CBT) - Futures Only (002602) Can be found here

Data has been downloaded and stored in \Data folder:

  • .\data\CHRIS-CME_C1.csv - Corn Futures Prices data
  • .\data\CFTC-002602_F_ALL.csv - Commitment of Traders data
In [1]:
import warnings
warnings.filterwarnings('ignore')
In [2]:
import pandas as pd
import numpy as np
from IPython.core.display import display, HTML
pd.options.display.max_colwidth = 500  # You need this, otherwise pandas
# will limit your HTML strings to 50 characters
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
pd.options.mode.chained_assignment = None  # default='warn'
from matplotlib import pyplot
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from math import sqrt
from numpy import concatenate
from sklearn.metrics import mean_squared_error
import matplotlib.pyplot as plt
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
import cufflinks as cf
import plotly.tools as tls
init_notebook_mode(connected=True)
cf.go_offline()
Using TensorFlow backend.
C:\Users\zilvi\Anaconda3\envs\zil_tensorflow\lib\site-packages\plotly\graph_objs\_deprecations.py:558: DeprecationWarning:

plotly.graph_objs.YAxis is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.layout.YAxis
  - plotly.graph_objs.layout.scene.YAxis


C:\Users\zilvi\Anaconda3\envs\zil_tensorflow\lib\site-packages\plotly\graph_objs\_deprecations.py:531: DeprecationWarning:

plotly.graph_objs.XAxis is deprecated.
Please replace it with one of the following more specific types
  - plotly.graph_objs.layout.XAxis
  - plotly.graph_objs.layout.scene.XAxis


2. Prepare and Explore Data

In [3]:
df_fut_orig = pd.read_csv('data\CHRIS-CME_C1.csv')
df_fut_orig.head(n=5)
Out[3]:
Date Open High Low Last Change Settle Volume Previous_Day_Open_Interest
0 2018-07-10 344.25 344.75 336.25 339.50 6.00 339.75 2668.0 2186.0
1 2018-07-09 346.00 348.50 342.50 346.00 6.00 345.75 3190.0 2969.0
2 2018-07-06 342.00 352.25 342.00 350.75 8.25 351.75 3068.0 3959.0
3 2018-07-05 345.50 348.75 341.50 342.50 0.75 343.50 3302.0 4812.0
4 2018-07-03 340.25 345.25 339.25 343.25 5.25 342.75 3048.0 5687.0
In [4]:
# Display a description of the dataset
display(df_fut_orig.describe())
Open High Low Last Change Settle Volume Previous_Day_Open_Interest
count 3033.000000 3034.000000 3034.000000 3034.000000 1081.000000 3034.000000 3034.000000 3034.00000
mean 457.095038 462.322924 451.795485 456.920040 3.950324 456.979318 103905.200396 352140.90145
std 140.338892 142.056030 138.436196 140.243019 3.415126 140.204571 73993.219920 248565.85531
min 219.000000 220.750000 216.750000 219.000000 0.000000 219.000000 0.000000 107.00000
25% 360.000000 363.000000 356.250000 359.500000 1.500000 359.750000 40172.750000 107559.25000
50% 388.500000 392.000000 383.500000 388.750000 3.000000 389.000000 102567.000000 365073.00000
75% 565.500000 573.562500 557.375000 564.625000 5.500000 564.625000 152391.250000 556408.50000
max 830.250000 843.750000 822.750000 831.250000 30.750000 831.250000 538170.000000 858696.00000
In [5]:
df_fut_orig['Date'] = pd.to_datetime(df_fut_orig['Date'])
df_fut_orig.set_index('Date',inplace=True)
df_fut_orig = df_fut_orig.sort_values('Date')

Plot Corn Futures Price Series using Plotly

In [6]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_original_price_series(df_fut_orig)

Seems there are some rows where Volume=0, lets find out more about these rows

In [7]:
df_fut_orig[df_fut_orig['Volume']<1]
Out[7]:
Open High Low Last Change Settle Volume Previous_Day_Open_Interest
Date
2007-04-05 359.75 367.50 357.25 366.00 NaN 366.00 0.0 354349.0
2012-04-06 658.25 658.25 658.25 658.25 NaN 658.25 0.0 401521.0
2015-04-03 386.50 386.50 386.50 386.50 NaN 386.50 0.0 470964.0

Since we will resample daily prices into weekly prices , lets drop those rows.

In [8]:
# drop outliers
df_fut_orig.drop(df_fut_orig[df_fut_orig.Volume<1].index, inplace=True)
In [9]:
df_cot_orig = pd.read_csv('data\CFTC-002602_F_ALL.csv')
display(df_cot_orig.head())
Date Open_Interest Producer_Merchant_Processor_User_Longs Producer_Merchant_Processor_User_Shorts Swap Dealer Longs Swap Dealer Shorts Swap Dealer Spreads Money Manager Longs Money Manager Shorts Money Manager Spreads Other Reportable Longs Other Reportable Shorts Other Reportable Spreads Total Reportable Longs Total Reportable Shorts Non Reportable Longs Non Reportable Shorts
0 2018-07-10 1818055.0 500172.0 750062.0 208128.0 39513.0 99477.0 263353.0 404297.0 154286.0 320946.0 70682.0 98709.0 1645071.0 1617026.0 172984.0 201029.0
1 2018-07-03 1830330.0 484257.0 773851.0 210341.0 36927.0 100340.0 274795.0 382191.0 149756.0 322256.0 66508.0 119627.0 1661372.0 1629200.0 168958.0 201130.0
2 2018-06-26 1885804.0 513100.0 840177.0 223131.0 32763.0 91972.0 287061.0 377825.0 153461.0 330396.0 58283.0 116745.0 1715866.0 1671226.0 169938.0 214578.0
3 2018-06-19 1992169.0 525197.0 920764.0 222105.0 41144.0 99285.0 299377.0 356828.0 163454.0 379025.0 56652.0 135078.0 1823521.0 1773205.0 168648.0 218964.0
4 2018-06-12 1963233.0 488666.0 917204.0 235249.0 37674.0 93281.0 292054.0 304292.0 172623.0 363918.0 65030.0 147098.0 1792889.0 1737202.0 170344.0 226031.0
In [10]:
display(df_cot_orig.describe())
Open_Interest Producer_Merchant_Processor_User_Longs Producer_Merchant_Processor_User_Shorts Swap Dealer Longs Swap Dealer Shorts Swap Dealer Spreads Money Manager Longs Money Manager Shorts Money Manager Spreads Other Reportable Longs Other Reportable Shorts Other Reportable Spreads Total Reportable Longs Total Reportable Shorts Non Reportable Longs Non Reportable Shorts
count 6.310000e+02 631.000000 6.310000e+02 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 631.000000 6.310000e+02 6.310000e+02 631.000000 631.000000
mean 1.292201e+06 270795.049128 6.268425e+05 290792.497623 20337.034865 33260.068146 236884.269414 137472.426307 94546.356577 140931.890650 70914.334390 85505.109350 1.152715e+06 1.068878e+06 139485.541997 223322.976228
std 2.095471e+05 68976.221600 1.554272e+05 53203.484072 18944.008732 22912.567257 67454.195123 109465.025186 32739.133163 51939.690903 26360.863384 29682.425476 1.939790e+05 2.060080e+05 23718.957966 29824.710288
min 7.482520e+05 102373.000000 2.972960e+05 186981.000000 0.000000 4397.000000 96989.000000 6714.000000 29130.000000 49809.000000 25905.000000 27592.000000 6.379810e+05 5.689510e+05 78578.000000 156086.000000
25% 1.192226e+06 226595.000000 5.235930e+05 255196.500000 6524.000000 13978.000000 186366.500000 47947.000000 72018.500000 104764.000000 53331.000000 62690.000000 1.055362e+06 9.573815e+05 121829.500000 198860.500000
50% 1.301506e+06 262823.000000 6.112810e+05 276337.000000 15239.000000 27209.000000 225682.000000 95548.000000 91850.000000 140343.000000 66261.000000 82705.000000 1.166372e+06 1.067548e+06 136966.000000 227337.000000
75% 1.398275e+06 314224.000000 7.058555e+05 321265.500000 28178.000000 48009.500000 287331.000000 211154.000000 113803.000000 175846.000000 83448.500000 106077.500000 1.247976e+06 1.180280e+06 153542.500000 246903.000000
max 1.992169e+06 525197.000000 1.001517e+06 422803.000000 95591.000000 113775.000000 431569.000000 447470.000000 231064.000000 379025.000000 173322.000000 181385.000000 1.825238e+06 1.773205e+06 206821.000000 293948.000000

Drop unnecessary columns columns and resample data

In [11]:
df_fut=df_fut_orig.drop(columns=[clmn for i,clmn in enumerate(df_fut_orig.columns) if i not in [5,6,7] ],axis=1)
display(df_fut.head())
Settle Volume Previous_Day_Open_Interest
Date
2006-06-16 235.50 56486.0 203491.0
2006-06-19 229.75 51299.0 190044.0
2006-06-20 229.75 41605.0 175859.0
2006-06-21 232.75 29803.0 162348.0
2006-06-22 230.50 28687.0 147658.0
In [12]:
s_settle =df_fut['Settle'].resample('W').last()
s_volume =df_fut['Volume'].resample('W').last()
df_fut_weekly = pd.concat([s_settle,s_volume], axis=1)
display(df_fut_weekly.head())
Settle Volume
Date
2006-06-18 235.50 56486.0
2006-06-25 228.25 28361.0
2006-07-02 235.50 30519.0
2006-07-09 241.00 13057.0
2006-07-16 253.50 2460.0
In [13]:
df_cot=df_cot_orig.drop(columns=[clmn for i,clmn in enumerate(df_cot_orig.columns) if i not in [0,1,2,3 ]],axis=1)
df_cot.rename(index=str, columns={"Producer_Merchant_Processor_User_Longs": "Longs", \
                                  "Producer_Merchant_Processor_User_Shorts": "Shorts"},inplace=True)
df_cot['Date'] = pd.to_datetime(df_cot['Date'])
df_cot.set_index('Date',inplace=True)
display(df_cot.head())
Open_Interest Longs Shorts
Date
2018-07-10 1818055.0 500172.0 750062.0
2018-07-03 1830330.0 484257.0 773851.0
2018-06-26 1885804.0 513100.0 840177.0
2018-06-19 1992169.0 525197.0 920764.0
2018-06-12 1963233.0 488666.0 917204.0
In [14]:
s_longs =df_cot['Longs'].resample('W').last()
s_shorts =df_cot['Shorts'].resample('W').last()
s_open_interest =df_cot['Open_Interest'].resample('W').last()
df_cot_weekly = pd.concat([s_open_interest,s_longs, s_shorts], axis=1)
display(df_cot_weekly.head(5))
Open_Interest Longs Shorts
Date
2006-06-18 1320155.0 209662.0 699163.0
2006-06-25 1321520.0 224476.0 666688.0
2006-07-02 1329400.0 234769.0 645735.0
2006-07-09 1327482.0 220552.0 648405.0
2006-07-16 1333225.0 216968.0 673110.0
In [15]:
df_weekly = pd.merge(df_fut_weekly,df_cot_weekly, on='Date')
display(df_weekly.head(5))
Settle Volume Open_Interest Longs Shorts
Date
2006-06-18 235.50 56486.0 1320155.0 209662.0 699163.0
2006-06-25 228.25 28361.0 1321520.0 224476.0 666688.0
2006-07-02 235.50 30519.0 1329400.0 234769.0 645735.0
2006-07-09 241.00 13057.0 1327482.0 220552.0 648405.0
2006-07-16 253.50 2460.0 1333225.0 216968.0 673110.0
In [16]:
# Display a description of the dataset
display(df_weekly.describe())
Settle Volume Open_Interest Longs Shorts
count 631.000000 631.000000 6.310000e+02 631.000000 6.310000e+02
mean 456.978605 100835.204437 1.292201e+06 270795.049128 6.268425e+05
std 140.242112 72466.341538 2.095471e+05 68976.221600 1.554272e+05
min 219.750000 132.000000 7.482520e+05 102373.000000 2.972960e+05
25% 359.500000 34822.500000 1.192226e+06 226595.000000 5.235930e+05
50% 389.250000 101209.000000 1.301506e+06 262823.000000 6.112810e+05
75% 560.375000 150341.000000 1.398275e+06 314224.000000 7.058555e+05
max 824.500000 369522.000000 1.992169e+06 525197.000000 1.001517e+06
In [17]:
# rest index since we need row numbers for splitting
df_weekly_idx_date=df_weekly.copy()
df_weekly.reset_index(inplace=True)

3. Visualise Data

In [18]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_weekly_combined_series_by_date(df_weekly)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [19]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_weekly_combined_series_by_trading_week(df_weekly)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [20]:
%load_ext autoreload
%autoreload 2
import visuals
visuals.plot_grouped_by_year_data(df_weekly_idx_date,"Stacked Plots of Price by Year")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [21]:
%load_ext autoreload
%autoreload 2
import visuals
visuals.lag_plot(df_weekly,"Lag Plot")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

4. Normalise the data using minmaxscaler

In [22]:
scaler = MinMaxScaler(feature_range=(0, 1))
values = df_weekly.loc[:, df_weekly.columns != 'Date'].values
scaled = scaler.fit_transform(values)

5. Split data into training, validation and test sets

In [23]:
validation_start=df_weekly[df_weekly['Date'] >= pd.to_datetime('2017-01-01')].index[0]
testing_start=df_weekly[df_weekly['Date'] >= pd.to_datetime('2018-01-01')].index[0]
In [24]:
print("validation start",validation_start)
print("testing start",testing_start)
validation start 550
testing start 603
In [25]:
# print data to double check
#print(df_weekly.iloc[validation_start])
#print(df_weekly.iloc[testing_start])
In [26]:
%load_ext autoreload
%autoreload 2
import data_preparer
reframed = data_preparer.series_to_supervised(scaled, 1, 1)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [27]:
# drop columns we don't want to predict
reframed.drop(reframed.columns[[6,7,8,9]], axis=1, inplace=True)
In [28]:
display(reframed.head())
var1(t-1) var2(t-1) var3(t-1) var4(t-1) var5(t-1) var1(t)
1 0.026044 0.152560 0.459760 0.253744 0.570655 0.014055
2 0.014055 0.076421 0.460857 0.288780 0.524540 0.026044
3 0.026044 0.082263 0.467192 0.313123 0.494786 0.035138
4 0.035138 0.034990 0.465650 0.279499 0.498578 0.055808
5 0.055808 0.006302 0.470267 0.271023 0.533659 0.028938

6. Define and Fit Model

In [29]:
%load_ext autoreload
%autoreload 2
import data_preparer
train_X, train_y, validation_X, validation_y,test_X, test_y = data_preparer.split_data(reframed,validation_start,testing_start)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [30]:
%load_ext autoreload
%autoreload 2
import models
model,history=models.basic_lstm_model(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Train on 550 samples, validate on 53 samples
Epoch 1/500
 - 2s - loss: 0.4313 - val_loss: 0.2146
Epoch 2/500
 - 0s - loss: 0.4074 - val_loss: 0.1917
Epoch 3/500
 - 0s - loss: 0.3848 - val_loss: 0.1699
Epoch 4/500
 - 0s - loss: 0.3635 - val_loss: 0.1487
Epoch 5/500
 - 0s - loss: 0.3429 - val_loss: 0.1279
Epoch 6/500
 - 0s - loss: 0.3228 - val_loss: 0.1071
Epoch 7/500
 - 0s - loss: 0.3029 - val_loss: 0.0861
Epoch 8/500
 - 0s - loss: 0.2831 - val_loss: 0.0645
Epoch 9/500
 - 0s - loss: 0.2633 - val_loss: 0.0425
Epoch 10/500
 - 0s - loss: 0.2448 - val_loss: 0.0266
Epoch 11/500
 - 0s - loss: 0.2285 - val_loss: 0.0210
Epoch 12/500
 - 0s - loss: 0.2166 - val_loss: 0.0248
Epoch 13/500
 - 0s - loss: 0.2085 - val_loss: 0.0335
Epoch 14/500
 - 0s - loss: 0.2028 - val_loss: 0.0442
Epoch 15/500
 - 0s - loss: 0.1991 - val_loss: 0.0528
Epoch 16/500
 - 0s - loss: 0.1967 - val_loss: 0.0593
Epoch 17/500
 - 0s - loss: 0.1950 - val_loss: 0.0643
Epoch 18/500
 - 0s - loss: 0.1937 - val_loss: 0.0681
Epoch 19/500
 - 0s - loss: 0.1927 - val_loss: 0.0711
Epoch 20/500
 - 0s - loss: 0.1918 - val_loss: 0.0740
Epoch 21/500
 - 0s - loss: 0.1910 - val_loss: 0.0769
Epoch 22/500
 - 0s - loss: 0.1901 - val_loss: 0.0796
Epoch 23/500
 - 0s - loss: 0.1894 - val_loss: 0.0820
Epoch 24/500
 - 0s - loss: 0.1886 - val_loss: 0.0843
Epoch 25/500
 - 0s - loss: 0.1880 - val_loss: 0.0864
Epoch 26/500
 - 0s - loss: 0.1873 - val_loss: 0.0885
Epoch 27/500
 - 0s - loss: 0.1867 - val_loss: 0.0900
Epoch 28/500
 - 0s - loss: 0.1861 - val_loss: 0.0913
Epoch 29/500
 - 0s - loss: 0.1856 - val_loss: 0.0922
Epoch 30/500
 - 0s - loss: 0.1852 - val_loss: 0.0932
Epoch 31/500
 - 0s - loss: 0.1847 - val_loss: 0.0937
Epoch 32/500
 - 0s - loss: 0.1843 - val_loss: 0.0942
Epoch 33/500
 - 0s - loss: 0.1838 - val_loss: 0.0945
Epoch 34/500
 - 0s - loss: 0.1834 - val_loss: 0.0948
Epoch 35/500
 - 0s - loss: 0.1830 - val_loss: 0.0951
Epoch 36/500
 - 0s - loss: 0.1826 - val_loss: 0.0954
Epoch 37/500
 - 0s - loss: 0.1822 - val_loss: 0.0957
Epoch 38/500
 - 0s - loss: 0.1818 - val_loss: 0.0958
Epoch 39/500
 - 0s - loss: 0.1814 - val_loss: 0.0960
Epoch 40/500
 - 0s - loss: 0.1810 - val_loss: 0.0962
Epoch 41/500
 - 0s - loss: 0.1806 - val_loss: 0.0964
Epoch 42/500
 - 0s - loss: 0.1801 - val_loss: 0.0967
Epoch 43/500
 - 0s - loss: 0.1797 - val_loss: 0.0969
Epoch 44/500
 - 0s - loss: 0.1793 - val_loss: 0.0972
Epoch 45/500
 - 0s - loss: 0.1789 - val_loss: 0.0975
Epoch 46/500
 - 0s - loss: 0.1785 - val_loss: 0.0979
Epoch 47/500
 - 0s - loss: 0.1781 - val_loss: 0.0982
Epoch 48/500
 - 0s - loss: 0.1777 - val_loss: 0.0986
Epoch 49/500
 - 0s - loss: 0.1773 - val_loss: 0.0989
Epoch 50/500
 - 0s - loss: 0.1768 - val_loss: 0.0992
Epoch 51/500
 - 0s - loss: 0.1764 - val_loss: 0.0995
Epoch 52/500
 - 0s - loss: 0.1760 - val_loss: 0.0997
Epoch 53/500
 - 0s - loss: 0.1756 - val_loss: 0.0999
Epoch 54/500
 - 0s - loss: 0.1752 - val_loss: 0.1001
Epoch 55/500
 - 0s - loss: 0.1748 - val_loss: 0.1003
Epoch 56/500
 - 0s - loss: 0.1743 - val_loss: 0.1004
Epoch 57/500
 - 0s - loss: 0.1739 - val_loss: 0.1006
Epoch 58/500
 - 0s - loss: 0.1735 - val_loss: 0.1007
Epoch 59/500
 - 0s - loss: 0.1731 - val_loss: 0.1007
Epoch 60/500
 - 0s - loss: 0.1726 - val_loss: 0.1008
Epoch 61/500
 - 0s - loss: 0.1722 - val_loss: 0.1008
Epoch 62/500
 - 0s - loss: 0.1718 - val_loss: 0.1009
Epoch 63/500
 - 0s - loss: 0.1714 - val_loss: 0.1010
Epoch 64/500
 - 0s - loss: 0.1709 - val_loss: 0.1012
Epoch 65/500
 - 0s - loss: 0.1705 - val_loss: 0.1012
Epoch 66/500
 - 0s - loss: 0.1700 - val_loss: 0.1013
Epoch 67/500
 - 0s - loss: 0.1696 - val_loss: 0.1013
Epoch 68/500
 - 0s - loss: 0.1691 - val_loss: 0.1014
Epoch 69/500
 - 0s - loss: 0.1687 - val_loss: 0.1016
Epoch 70/500
 - 0s - loss: 0.1682 - val_loss: 0.1018
Epoch 71/500
 - 0s - loss: 0.1677 - val_loss: 0.1020
Epoch 72/500
 - 0s - loss: 0.1673 - val_loss: 0.1022
Epoch 73/500
 - 0s - loss: 0.1668 - val_loss: 0.1024
Epoch 74/500
 - 0s - loss: 0.1663 - val_loss: 0.1026
Epoch 75/500
 - 0s - loss: 0.1658 - val_loss: 0.1029
Epoch 76/500
 - 0s - loss: 0.1653 - val_loss: 0.1031
Epoch 77/500
 - 0s - loss: 0.1649 - val_loss: 0.1033
Epoch 78/500
 - 0s - loss: 0.1644 - val_loss: 0.1034
Epoch 79/500
 - 0s - loss: 0.1639 - val_loss: 0.1036
Epoch 80/500
 - 0s - loss: 0.1634 - val_loss: 0.1037
Epoch 81/500
 - 0s - loss: 0.1628 - val_loss: 0.1039
Epoch 82/500
 - 0s - loss: 0.1623 - val_loss: 0.1040
Epoch 83/500
 - 0s - loss: 0.1618 - val_loss: 0.1042
Epoch 84/500
 - 0s - loss: 0.1613 - val_loss: 0.1044
Epoch 85/500
 - 0s - loss: 0.1608 - val_loss: 0.1044
Epoch 86/500
 - 0s - loss: 0.1603 - val_loss: 0.1045
Epoch 87/500
 - 0s - loss: 0.1597 - val_loss: 0.1046
Epoch 88/500
 - 0s - loss: 0.1592 - val_loss: 0.1047
Epoch 89/500
 - 0s - loss: 0.1587 - val_loss: 0.1048
Epoch 90/500
 - 0s - loss: 0.1581 - val_loss: 0.1049
Epoch 91/500
 - 0s - loss: 0.1575 - val_loss: 0.1050
Epoch 92/500
 - 0s - loss: 0.1570 - val_loss: 0.1049
Epoch 93/500
 - 0s - loss: 0.1564 - val_loss: 0.1049
Epoch 94/500
 - 0s - loss: 0.1559 - val_loss: 0.1050
Epoch 95/500
 - 0s - loss: 0.1553 - val_loss: 0.1050
Epoch 96/500
 - 0s - loss: 0.1547 - val_loss: 0.1050
Epoch 97/500
 - 0s - loss: 0.1542 - val_loss: 0.1050
Epoch 98/500
 - 0s - loss: 0.1536 - val_loss: 0.1051
Epoch 99/500
 - 0s - loss: 0.1530 - val_loss: 0.1052
Epoch 100/500
 - 0s - loss: 0.1524 - val_loss: 0.1052
Epoch 101/500
 - 0s - loss: 0.1518 - val_loss: 0.1052
Epoch 102/500
 - 0s - loss: 0.1512 - val_loss: 0.1052
Epoch 103/500
 - 0s - loss: 0.1506 - val_loss: 0.1052
Epoch 104/500
 - 0s - loss: 0.1500 - val_loss: 0.1052
Epoch 105/500
 - 0s - loss: 0.1494 - val_loss: 0.1051
Epoch 106/500
 - 0s - loss: 0.1488 - val_loss: 0.1051
Epoch 107/500
 - 0s - loss: 0.1482 - val_loss: 0.1051
Epoch 108/500
 - 0s - loss: 0.1475 - val_loss: 0.1052
Epoch 109/500
 - 0s - loss: 0.1469 - val_loss: 0.1052
Epoch 110/500
 - 0s - loss: 0.1462 - val_loss: 0.1052
Epoch 111/500
 - 0s - loss: 0.1455 - val_loss: 0.1053
Epoch 112/500
 - 0s - loss: 0.1449 - val_loss: 0.1055
Epoch 113/500
 - 0s - loss: 0.1442 - val_loss: 0.1056
Epoch 114/500
 - 0s - loss: 0.1435 - val_loss: 0.1058
Epoch 115/500
 - 0s - loss: 0.1428 - val_loss: 0.1059
Epoch 116/500
 - 0s - loss: 0.1421 - val_loss: 0.1060
Epoch 117/500
 - 0s - loss: 0.1414 - val_loss: 0.1059
Epoch 118/500
 - 0s - loss: 0.1407 - val_loss: 0.1057
Epoch 119/500
 - 0s - loss: 0.1400 - val_loss: 0.1056
Epoch 120/500
 - 0s - loss: 0.1392 - val_loss: 0.1054
Epoch 121/500
 - 0s - loss: 0.1385 - val_loss: 0.1054
Epoch 122/500
 - 0s - loss: 0.1378 - val_loss: 0.1055
Epoch 123/500
 - 0s - loss: 0.1370 - val_loss: 0.1055
Epoch 124/500
 - 0s - loss: 0.1363 - val_loss: 0.1056
Epoch 125/500
 - 0s - loss: 0.1355 - val_loss: 0.1057
Epoch 126/500
 - 0s - loss: 0.1347 - val_loss: 0.1058
Epoch 127/500
 - 0s - loss: 0.1340 - val_loss: 0.1059
Epoch 128/500
 - 0s - loss: 0.1332 - val_loss: 0.1059
Epoch 129/500
 - 0s - loss: 0.1324 - val_loss: 0.1060
Epoch 130/500
 - 0s - loss: 0.1316 - val_loss: 0.1060
Epoch 131/500
 - 0s - loss: 0.1308 - val_loss: 0.1059
Epoch 132/500
 - 0s - loss: 0.1301 - val_loss: 0.1058
Epoch 133/500
 - 0s - loss: 0.1293 - val_loss: 0.1060
Epoch 134/500
 - 0s - loss: 0.1284 - val_loss: 0.1061
Epoch 135/500
 - 0s - loss: 0.1276 - val_loss: 0.1061
Epoch 136/500
 - 0s - loss: 0.1268 - val_loss: 0.1060
Epoch 137/500
 - 0s - loss: 0.1260 - val_loss: 0.1060
Epoch 138/500
 - 0s - loss: 0.1252 - val_loss: 0.1058
Epoch 139/500
 - 0s - loss: 0.1243 - val_loss: 0.1056
Epoch 140/500
 - 0s - loss: 0.1235 - val_loss: 0.1054
Epoch 141/500
 - 0s - loss: 0.1226 - val_loss: 0.1052
Epoch 142/500
 - 0s - loss: 0.1218 - val_loss: 0.1049
Epoch 143/500
 - 0s - loss: 0.1209 - val_loss: 0.1045
Epoch 144/500
 - 0s - loss: 0.1200 - val_loss: 0.1041
Epoch 145/500
 - 0s - loss: 0.1192 - val_loss: 0.1036
Epoch 146/500
 - 0s - loss: 0.1183 - val_loss: 0.1030
Epoch 147/500
 - 0s - loss: 0.1174 - val_loss: 0.1026
Epoch 148/500
 - 0s - loss: 0.1165 - val_loss: 0.1023
Epoch 149/500
 - 0s - loss: 0.1155 - val_loss: 0.1020
Epoch 150/500
 - 0s - loss: 0.1146 - val_loss: 0.1017
Epoch 151/500
 - 0s - loss: 0.1136 - val_loss: 0.1013
Epoch 152/500
 - 0s - loss: 0.1127 - val_loss: 0.1008
Epoch 153/500
 - 0s - loss: 0.1117 - val_loss: 0.1002
Epoch 154/500
 - 0s - loss: 0.1107 - val_loss: 0.0996
Epoch 155/500
 - 0s - loss: 0.1097 - val_loss: 0.0989
Epoch 156/500
 - 0s - loss: 0.1087 - val_loss: 0.0982
Epoch 157/500
 - 0s - loss: 0.1076 - val_loss: 0.0975
Epoch 158/500
 - 0s - loss: 0.1066 - val_loss: 0.0966
Epoch 159/500
 - 0s - loss: 0.1056 - val_loss: 0.0960
Epoch 160/500
 - 0s - loss: 0.1045 - val_loss: 0.0955
Epoch 161/500
 - 0s - loss: 0.1034 - val_loss: 0.0948
Epoch 162/500
 - 0s - loss: 0.1023 - val_loss: 0.0939
Epoch 163/500
 - 0s - loss: 0.1012 - val_loss: 0.0930
Epoch 164/500
 - 0s - loss: 0.1001 - val_loss: 0.0920
Epoch 165/500
 - 0s - loss: 0.0990 - val_loss: 0.0908
Epoch 166/500
 - 0s - loss: 0.0979 - val_loss: 0.0894
Epoch 167/500
 - 0s - loss: 0.0967 - val_loss: 0.0885
Epoch 168/500
 - 0s - loss: 0.0956 - val_loss: 0.0878
Epoch 169/500
 - 0s - loss: 0.0944 - val_loss: 0.0869
Epoch 170/500
 - 0s - loss: 0.0933 - val_loss: 0.0860
Epoch 171/500
 - 0s - loss: 0.0921 - val_loss: 0.0852
Epoch 172/500
 - 0s - loss: 0.0908 - val_loss: 0.0845
Epoch 173/500
 - 0s - loss: 0.0896 - val_loss: 0.0837
Epoch 174/500
 - 0s - loss: 0.0883 - val_loss: 0.0827
Epoch 175/500
 - 0s - loss: 0.0871 - val_loss: 0.0812
Epoch 176/500
 - 0s - loss: 0.0859 - val_loss: 0.0797
Epoch 177/500
 - 0s - loss: 0.0846 - val_loss: 0.0785
Epoch 178/500
 - 0s - loss: 0.0833 - val_loss: 0.0772
Epoch 179/500
 - 0s - loss: 0.0820 - val_loss: 0.0760
Epoch 180/500
 - 0s - loss: 0.0807 - val_loss: 0.0747
Epoch 181/500
 - 0s - loss: 0.0794 - val_loss: 0.0736
Epoch 182/500
 - 0s - loss: 0.0781 - val_loss: 0.0725
Epoch 183/500
 - 0s - loss: 0.0767 - val_loss: 0.0713
Epoch 184/500
 - 0s - loss: 0.0753 - val_loss: 0.0698
Epoch 185/500
 - 0s - loss: 0.0740 - val_loss: 0.0682
Epoch 186/500
 - 0s - loss: 0.0726 - val_loss: 0.0666
Epoch 187/500
 - 0s - loss: 0.0713 - val_loss: 0.0649
Epoch 188/500
 - 0s - loss: 0.0699 - val_loss: 0.0630
Epoch 189/500
 - 0s - loss: 0.0685 - val_loss: 0.0609
Epoch 190/500
 - 0s - loss: 0.0671 - val_loss: 0.0587
Epoch 191/500
 - 0s - loss: 0.0657 - val_loss: 0.0565
Epoch 192/500
 - 0s - loss: 0.0644 - val_loss: 0.0543
Epoch 193/500
 - 0s - loss: 0.0630 - val_loss: 0.0520
Epoch 194/500
 - 0s - loss: 0.0616 - val_loss: 0.0500
Epoch 195/500
 - 0s - loss: 0.0602 - val_loss: 0.0479
Epoch 196/500
 - 0s - loss: 0.0589 - val_loss: 0.0463
Epoch 197/500
 - 0s - loss: 0.0574 - val_loss: 0.0449
Epoch 198/500
 - 0s - loss: 0.0560 - val_loss: 0.0427
Epoch 199/500
 - 0s - loss: 0.0547 - val_loss: 0.0407
Epoch 200/500
 - 0s - loss: 0.0534 - val_loss: 0.0390
Epoch 201/500
 - 0s - loss: 0.0521 - val_loss: 0.0378
Epoch 202/500
 - 0s - loss: 0.0507 - val_loss: 0.0365
Epoch 203/500
 - 0s - loss: 0.0495 - val_loss: 0.0354
Epoch 204/500
 - 0s - loss: 0.0482 - val_loss: 0.0343
Epoch 205/500
 - 0s - loss: 0.0469 - val_loss: 0.0331
Epoch 206/500
 - 0s - loss: 0.0458 - val_loss: 0.0316
Epoch 207/500
 - 0s - loss: 0.0446 - val_loss: 0.0306
Epoch 208/500
 - 0s - loss: 0.0434 - val_loss: 0.0295
Epoch 209/500
 - 0s - loss: 0.0422 - val_loss: 0.0281
Epoch 210/500
 - 0s - loss: 0.0412 - val_loss: 0.0266
Epoch 211/500
 - 0s - loss: 0.0401 - val_loss: 0.0250
Epoch 212/500
 - 0s - loss: 0.0392 - val_loss: 0.0234
Epoch 213/500
 - 0s - loss: 0.0383 - val_loss: 0.0221
Epoch 214/500
 - 0s - loss: 0.0374 - val_loss: 0.0212
Epoch 215/500
 - 0s - loss: 0.0365 - val_loss: 0.0198
Epoch 216/500
 - 0s - loss: 0.0356 - val_loss: 0.0184
Epoch 217/500
 - 0s - loss: 0.0349 - val_loss: 0.0175
Epoch 218/500
 - 0s - loss: 0.0341 - val_loss: 0.0168
Epoch 219/500
 - 0s - loss: 0.0335 - val_loss: 0.0161
Epoch 220/500
 - 0s - loss: 0.0329 - val_loss: 0.0155
Epoch 221/500
 - 0s - loss: 0.0324 - val_loss: 0.0149
Epoch 222/500
 - 0s - loss: 0.0319 - val_loss: 0.0145
Epoch 223/500
 - 0s - loss: 0.0315 - val_loss: 0.0142
Epoch 224/500
 - 0s - loss: 0.0312 - val_loss: 0.0138
Epoch 225/500
 - 0s - loss: 0.0309 - val_loss: 0.0133
Epoch 226/500
 - 0s - loss: 0.0306 - val_loss: 0.0130
Epoch 227/500
 - 0s - loss: 0.0304 - val_loss: 0.0127
Epoch 228/500
 - 0s - loss: 0.0302 - val_loss: 0.0125
Epoch 229/500
 - 0s - loss: 0.0300 - val_loss: 0.0123
Epoch 230/500
 - 0s - loss: 0.0298 - val_loss: 0.0121
Epoch 231/500
 - 0s - loss: 0.0297 - val_loss: 0.0120
Epoch 232/500
 - 0s - loss: 0.0296 - val_loss: 0.0119
Epoch 233/500
 - 0s - loss: 0.0295 - val_loss: 0.0118
Epoch 234/500
 - 0s - loss: 0.0294 - val_loss: 0.0117
Epoch 235/500
 - 0s - loss: 0.0293 - val_loss: 0.0116
Epoch 236/500
 - 0s - loss: 0.0292 - val_loss: 0.0115
Epoch 237/500
 - 0s - loss: 0.0291 - val_loss: 0.0115
Epoch 238/500
 - 0s - loss: 0.0291 - val_loss: 0.0114
Epoch 239/500
 - 0s - loss: 0.0290 - val_loss: 0.0114
Epoch 240/500
 - 0s - loss: 0.0290 - val_loss: 0.0113
Epoch 241/500
 - 0s - loss: 0.0289 - val_loss: 0.0113
Epoch 242/500
 - 0s - loss: 0.0289 - val_loss: 0.0112
Epoch 243/500
 - 0s - loss: 0.0288 - val_loss: 0.0112
Epoch 244/500
 - 0s - loss: 0.0288 - val_loss: 0.0112
Epoch 245/500
 - 0s - loss: 0.0287 - val_loss: 0.0112
Epoch 246/500
 - 0s - loss: 0.0287 - val_loss: 0.0112
Epoch 247/500
 - 0s - loss: 0.0286 - val_loss: 0.0111
Epoch 248/500
 - 0s - loss: 0.0286 - val_loss: 0.0111
Epoch 249/500
 - 0s - loss: 0.0285 - val_loss: 0.0111
Epoch 250/500
 - 0s - loss: 0.0285 - val_loss: 0.0111
Epoch 251/500
 - 0s - loss: 0.0284 - val_loss: 0.0111
Epoch 252/500
 - 0s - loss: 0.0284 - val_loss: 0.0111
Epoch 253/500
 - 0s - loss: 0.0283 - val_loss: 0.0111
Epoch 254/500
 - 0s - loss: 0.0284 - val_loss: 0.0111
Epoch 255/500
 - 0s - loss: 0.0282 - val_loss: 0.0110
Epoch 256/500
 - 0s - loss: 0.0283 - val_loss: 0.0110
Epoch 257/500
 - 0s - loss: 0.0282 - val_loss: 0.0111
Epoch 258/500
 - 0s - loss: 0.0282 - val_loss: 0.0110
Epoch 259/500
 - 0s - loss: 0.0281 - val_loss: 0.0111
Epoch 260/500
 - 0s - loss: 0.0282 - val_loss: 0.0110
Epoch 261/500
 - 0s - loss: 0.0280 - val_loss: 0.0110
Epoch 262/500
 - 0s - loss: 0.0281 - val_loss: 0.0110
Epoch 263/500
 - 0s - loss: 0.0280 - val_loss: 0.0110
Epoch 264/500
 - 0s - loss: 0.0280 - val_loss: 0.0110
Epoch 265/500
 - 0s - loss: 0.0279 - val_loss: 0.0110
Epoch 266/500
 - 0s - loss: 0.0280 - val_loss: 0.0110
Epoch 267/500
 - 0s - loss: 0.0279 - val_loss: 0.0110
Epoch 268/500
 - 0s - loss: 0.0280 - val_loss: 0.0110
Epoch 269/500
 - 0s - loss: 0.0279 - val_loss: 0.0110
Epoch 270/500
 - 0s - loss: 0.0279 - val_loss: 0.0110
Epoch 271/500
 - 0s - loss: 0.0278 - val_loss: 0.0110
Epoch 272/500
 - 0s - loss: 0.0279 - val_loss: 0.0110
Epoch 273/500
 - 0s - loss: 0.0278 - val_loss: 0.0110
Epoch 274/500
 - 0s - loss: 0.0278 - val_loss: 0.0110
Epoch 275/500
 - 0s - loss: 0.0278 - val_loss: 0.0109
Epoch 276/500
 - 0s - loss: 0.0278 - val_loss: 0.0110
Epoch 277/500
 - 0s - loss: 0.0277 - val_loss: 0.0110
Epoch 278/500
 - 0s - loss: 0.0278 - val_loss: 0.0110
Epoch 279/500
 - 0s - loss: 0.0277 - val_loss: 0.0111
Epoch 280/500
 - 0s - loss: 0.0277 - val_loss: 0.0110
Epoch 281/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 282/500
 - 0s - loss: 0.0277 - val_loss: 0.0110
Epoch 283/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 284/500
 - 0s - loss: 0.0276 - val_loss: 0.0109
Epoch 285/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 286/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 287/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 288/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 289/500
 - 0s - loss: 0.0275 - val_loss: 0.0111
Epoch 290/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 291/500
 - 0s - loss: 0.0275 - val_loss: 0.0112
Epoch 292/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 293/500
 - 0s - loss: 0.0275 - val_loss: 0.0111
Epoch 294/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 295/500
 - 0s - loss: 0.0275 - val_loss: 0.0111
Epoch 296/500
 - 0s - loss: 0.0276 - val_loss: 0.0110
Epoch 297/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 298/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 299/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 300/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 301/500
 - 0s - loss: 0.0275 - val_loss: 0.0111
Epoch 302/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 303/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 304/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 305/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 306/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 307/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 308/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 309/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 310/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 311/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 312/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 313/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 314/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 315/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 316/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 317/500
 - 0s - loss: 0.0274 - val_loss: 0.0111
Epoch 318/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 319/500
 - 0s - loss: 0.0274 - val_loss: 0.0111
Epoch 320/500
 - 0s - loss: 0.0275 - val_loss: 0.0110
Epoch 321/500
 - 0s - loss: 0.0273 - val_loss: 0.0111
Epoch 322/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 323/500
 - 0s - loss: 0.0273 - val_loss: 0.0111
Epoch 324/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 325/500
 - 0s - loss: 0.0273 - val_loss: 0.0111
Epoch 326/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 327/500
 - 0s - loss: 0.0273 - val_loss: 0.0111
Epoch 328/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 329/500
 - 0s - loss: 0.0273 - val_loss: 0.0111
Epoch 330/500
 - 0s - loss: 0.0274 - val_loss: 0.0110
Epoch 331/500
 - 0s - loss: 0.0273 - val_loss: 0.0111
Epoch 332/500
 - 0s - loss: 0.0273 - val_loss: 0.0109
Epoch 333/500
 - 0s - loss: 0.0272 - val_loss: 0.0110
Epoch 334/500
 - 0s - loss: 0.0272 - val_loss: 0.0110
Epoch 335/500
 - 0s - loss: 0.0272 - val_loss: 0.0110
Epoch 336/500
 - 0s - loss: 0.0273 - val_loss: 0.0109
Epoch 337/500
 - 0s - loss: 0.0272 - val_loss: 0.0110
Epoch 338/500
 - 0s - loss: 0.0272 - val_loss: 0.0109
Epoch 339/500
 - 0s - loss: 0.0272 - val_loss: 0.0111
Epoch 340/500
 - 0s - loss: 0.0272 - val_loss: 0.0109
Epoch 341/500
 - 0s - loss: 0.0271 - val_loss: 0.0110
Epoch 342/500
 - 0s - loss: 0.0272 - val_loss: 0.0109
Epoch 343/500
 - 0s - loss: 0.0271 - val_loss: 0.0110
Epoch 344/500
 - 0s - loss: 0.0272 - val_loss: 0.0109
Epoch 345/500
 - 0s - loss: 0.0271 - val_loss: 0.0110
Epoch 346/500
 - 0s - loss: 0.0271 - val_loss: 0.0109
Epoch 347/500
 - 0s - loss: 0.0271 - val_loss: 0.0110
Epoch 348/500
 - 0s - loss: 0.0271 - val_loss: 0.0109
Epoch 349/500
 - 0s - loss: 0.0271 - val_loss: 0.0110
Epoch 350/500
 - 0s - loss: 0.0271 - val_loss: 0.0109
Epoch 351/500
 - 0s - loss: 0.0271 - val_loss: 0.0111
Epoch 352/500
 - 0s - loss: 0.0271 - val_loss: 0.0109
Epoch 353/500
 - 0s - loss: 0.0271 - val_loss: 0.0110
Epoch 354/500
 - 0s - loss: 0.0271 - val_loss: 0.0110
Epoch 355/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 356/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 357/500
 - 0s - loss: 0.0271 - val_loss: 0.0109
Epoch 358/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 359/500
 - 0s - loss: 0.0271 - val_loss: 0.0109
Epoch 360/500
 - 0s - loss: 0.0270 - val_loss: 0.0111
Epoch 361/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 362/500
 - 0s - loss: 0.0270 - val_loss: 0.0111
Epoch 363/500
 - 0s - loss: 0.0270 - val_loss: 0.0111
Epoch 364/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 365/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 366/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 367/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 368/500
 - 0s - loss: 0.0270 - val_loss: 0.0111
Epoch 369/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 370/500
 - 0s - loss: 0.0270 - val_loss: 0.0111
Epoch 371/500
 - 0s - loss: 0.0270 - val_loss: 0.0111
Epoch 372/500
 - 0s - loss: 0.0270 - val_loss: 0.0111
Epoch 373/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 374/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 375/500
 - 0s - loss: 0.0270 - val_loss: 0.0110
Epoch 376/500
 - 0s - loss: 0.0269 - val_loss: 0.0110
Epoch 377/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 378/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 379/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 380/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 381/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 382/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 383/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 384/500
 - 0s - loss: 0.0269 - val_loss: 0.0110
Epoch 385/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 386/500
 - 0s - loss: 0.0269 - val_loss: 0.0110
Epoch 387/500
 - 0s - loss: 0.0269 - val_loss: 0.0111
Epoch 388/500
 - 0s - loss: 0.0269 - val_loss: 0.0110
Epoch 389/500
 - 0s - loss: 0.0269 - val_loss: 0.0110
Epoch 390/500
 - 0s - loss: 0.0269 - val_loss: 0.0112
Epoch 391/500
 - 0s - loss: 0.0269 - val_loss: 0.0110
Epoch 392/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 393/500
 - 0s - loss: 0.0268 - val_loss: 0.0111
Epoch 394/500
 - 0s - loss: 0.0269 - val_loss: 0.0110
Epoch 395/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 396/500
 - 0s - loss: 0.0268 - val_loss: 0.0111
Epoch 397/500
 - 0s - loss: 0.0269 - val_loss: 0.0110
Epoch 398/500
 - 0s - loss: 0.0268 - val_loss: 0.0109
Epoch 399/500
 - 0s - loss: 0.0268 - val_loss: 0.0111
Epoch 400/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 401/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 402/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 403/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 404/500
 - 0s - loss: 0.0268 - val_loss: 0.0111
Epoch 405/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 406/500
 - 0s - loss: 0.0268 - val_loss: 0.0111
Epoch 407/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 408/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 409/500
 - 0s - loss: 0.0268 - val_loss: 0.0111
Epoch 410/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 411/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 412/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 413/500
 - 0s - loss: 0.0268 - val_loss: 0.0111
Epoch 414/500
 - 0s - loss: 0.0268 - val_loss: 0.0110
Epoch 415/500
 - 0s - loss: 0.0267 - val_loss: 0.0109
Epoch 416/500
 - 0s - loss: 0.0267 - val_loss: 0.0111
Epoch 417/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 418/500
 - 0s - loss: 0.0267 - val_loss: 0.0111
Epoch 419/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 420/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 421/500
 - 0s - loss: 0.0267 - val_loss: 0.0111
Epoch 422/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 423/500
 - 0s - loss: 0.0267 - val_loss: 0.0109
Epoch 424/500
 - 0s - loss: 0.0267 - val_loss: 0.0111
Epoch 425/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 426/500
 - 0s - loss: 0.0267 - val_loss: 0.0109
Epoch 427/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 428/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 429/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 430/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 431/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 432/500
 - 0s - loss: 0.0267 - val_loss: 0.0109
Epoch 433/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 434/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 435/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 436/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 437/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 438/500
 - 0s - loss: 0.0267 - val_loss: 0.0109
Epoch 439/500
 - 0s - loss: 0.0266 - val_loss: 0.0111
Epoch 440/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 441/500
 - 0s - loss: 0.0267 - val_loss: 0.0110
Epoch 442/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 443/500
 - 0s - loss: 0.0266 - val_loss: 0.0111
Epoch 444/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 445/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 446/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 447/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 448/500
 - 0s - loss: 0.0266 - val_loss: 0.0109
Epoch 449/500
 - 0s - loss: 0.0266 - val_loss: 0.0111
Epoch 450/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 451/500
 - 0s - loss: 0.0266 - val_loss: 0.0109
Epoch 452/500
 - 0s - loss: 0.0266 - val_loss: 0.0112
Epoch 453/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 454/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 455/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 456/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 457/500
 - 0s - loss: 0.0266 - val_loss: 0.0111
Epoch 458/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 459/500
 - 0s - loss: 0.0266 - val_loss: 0.0109
Epoch 460/500
 - 0s - loss: 0.0266 - val_loss: 0.0111
Epoch 461/500
 - 0s - loss: 0.0266 - val_loss: 0.0111
Epoch 462/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 463/500
 - 0s - loss: 0.0266 - val_loss: 0.0111
Epoch 464/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 465/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 466/500
 - 0s - loss: 0.0266 - val_loss: 0.0111
Epoch 467/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 468/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 469/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 470/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 471/500
 - 0s - loss: 0.0266 - val_loss: 0.0110
Epoch 472/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 473/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 474/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 475/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 476/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 477/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 478/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 479/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 480/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 481/500
 - 0s - loss: 0.0265 - val_loss: 0.0109
Epoch 482/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 483/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 484/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 485/500
 - 0s - loss: 0.0265 - val_loss: 0.0109
Epoch 486/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 487/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 488/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 489/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 490/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 491/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 492/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 493/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 494/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 495/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 496/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 497/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 498/500
 - 0s - loss: 0.0265 - val_loss: 0.0111
Epoch 499/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
Epoch 500/500
 - 0s - loss: 0.0265 - val_loss: 0.0110
In [31]:
pyplot.plot(history['loss'], label='train')
pyplot.plot(history['val_loss'], label='validation')
pyplot.legend()
pyplot.show()
In [32]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(validation_X,validation_y,model,scaler)
print('LSTM Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Model on Validation Data RMSE: 8.663
In [33]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_series_to_compare(inv_y,inv_yhat,"Actual Price","Predicted Price", "Actual Price Versus LSTM Predicted Price")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

7. Bench Mark Model

In this section we will check our bench mark model. As is proposed in my proposal my bench mark model is a simple linear regressor model.

Load the preprocessed data

In [34]:
from pandas import read_csv
from pandas import datetime
from pandas import DataFrame
from pandas import concat
from matplotlib import pyplot
from sklearn.metrics import mean_squared_error
from math import sqrt


# Create lagged dataset
values = pd.DataFrame(df_weekly["Settle"].values)
df_benchmark  = concat([values.shift(1), values], axis=1)
df_benchmark.columns = ['t', 't+1']
display(df_benchmark.head(5))
t t+1
0 NaN 235.50
1 235.50 228.25
2 228.25 235.50
3 235.50 241.00
4 241.00 253.50
In [35]:
# split into train , validation and test sets
X = df_benchmark.values
train, validation, test = X[1:validation_start], X[validation_start:testing_start],X[testing_start:]
train_bench_X, train_bench_y = train[:,0], train[:,1]
validation_bench_X, validation_bench_y = validation[:,0], validation[:,1]
test_bench_X, test_bench_y = test[:,0], test[:,1]
In [36]:
%load_ext autoreload
%autoreload 2
import models
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [37]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
predictions,rmse=models.make_benchmark_model_prediction(validation_bench_X,validation_bench_y)
print('Benchmark Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Benchmark Model on Validation Data RMSE: 8.750
In [38]:
%load_ext autoreload
%autoreload 2
import visuals

visuals.plot_series_to_compare(validation_bench_y,predictions,"Actual Price","Predicted Price", "Actual Price Versus Benchmark Model Predicted Price")
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

8. Test model on unseen data

Test LSTM model on unseen data

In [39]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(test_X,test_y,model,scaler)
print('LSTM Moddel on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Moddel on Test Data RMSE: 8.950

Test Benchmark model on unseen data

In [40]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
predictions,rmse=models.make_benchmark_model_prediction(test_bench_X,test_bench_y)
print('Benchmark Model on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Benchmark Model on Test Data RMSE: 8.293

9. Tune basic LSTM Model

In [41]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_memmory_cells(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=1.000000, loss=0.012129
>2/5 param=1.000000, loss=0.014510
>3/5 param=1.000000, loss=0.011251
>4/5 param=1.000000, loss=0.013165
>5/5 param=1.000000, loss=0.012103
>1/5 param=5.000000, loss=0.011613
>2/5 param=5.000000, loss=0.012066
>3/5 param=5.000000, loss=0.011987
>4/5 param=5.000000, loss=0.012024
>5/5 param=5.000000, loss=0.012577
>1/5 param=10.000000, loss=0.012330
>2/5 param=10.000000, loss=0.013115
>3/5 param=10.000000, loss=0.013052
>4/5 param=10.000000, loss=0.011792
>5/5 param=10.000000, loss=0.013219
>1/5 param=25.000000, loss=0.011451
>2/5 param=25.000000, loss=0.013046
>3/5 param=25.000000, loss=0.011217
>4/5 param=25.000000, loss=0.011381
>5/5 param=25.000000, loss=0.011058
>1/5 param=50.000000, loss=0.012644
>2/5 param=50.000000, loss=0.012646
>3/5 param=50.000000, loss=0.011140
>4/5 param=50.000000, loss=0.013345
>5/5 param=50.000000, loss=0.012515
>1/5 param=100.000000, loss=0.011604
>2/5 param=100.000000, loss=0.015141
>3/5 param=100.000000, loss=0.012493
>4/5 param=100.000000, loss=0.012222
>5/5 param=100.000000, loss=0.013316
>1/5 param=200.000000, loss=0.011432
>2/5 param=200.000000, loss=0.012943
>3/5 param=200.000000, loss=0.010766
>4/5 param=200.000000, loss=0.014985
>5/5 param=200.000000, loss=0.011893
              1         5        10        25        50       100       200
count  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000
mean   0.012632  0.012053  0.012702  0.011631  0.012458  0.012955  0.012404
std    0.001250  0.000344  0.000618  0.000806  0.000805  0.001368  0.001646
min    0.011251  0.011613  0.011792  0.011058  0.011140  0.011604  0.010766
25%    0.012103  0.011987  0.012330  0.011217  0.012515  0.012222  0.011432
50%    0.012129  0.012024  0.013052  0.011381  0.012644  0.012493  0.011893
75%    0.013165  0.012066  0.013115  0.011451  0.012646  0.013316  0.012943
max    0.014510  0.012577  0.013219  0.013046  0.013345  0.015141  0.014985
In [42]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_batch_size(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=2.000000, loss=0.017062
>2/5 param=2.000000, loss=0.017288
>3/5 param=2.000000, loss=0.019628
>4/5 param=2.000000, loss=0.017816
>5/5 param=2.000000, loss=0.019510
>1/5 param=4.000000, loss=0.012946
>2/5 param=4.000000, loss=0.013468
>3/5 param=4.000000, loss=0.012065
>4/5 param=4.000000, loss=0.012588
>5/5 param=4.000000, loss=0.013342
>1/5 param=8.000000, loss=0.014913
>2/5 param=8.000000, loss=0.016104
>3/5 param=8.000000, loss=0.015724
>4/5 param=8.000000, loss=0.015701
>5/5 param=8.000000, loss=0.014112
>1/5 param=32.000000, loss=0.011369
>2/5 param=32.000000, loss=0.011775
>3/5 param=32.000000, loss=0.012824
>4/5 param=32.000000, loss=0.012704
>5/5 param=32.000000, loss=0.011133
>1/5 param=64.000000, loss=0.011609
>2/5 param=64.000000, loss=0.011532
>3/5 param=64.000000, loss=0.013435
>4/5 param=64.000000, loss=0.011951
>5/5 param=64.000000, loss=0.012349
>1/5 param=128.000000, loss=0.011928
>2/5 param=128.000000, loss=0.012988
>3/5 param=128.000000, loss=0.011940
>4/5 param=128.000000, loss=0.011974
>5/5 param=128.000000, loss=0.011488
>1/5 param=256.000000, loss=0.011780
>2/5 param=256.000000, loss=0.013215
>3/5 param=256.000000, loss=0.012355
>4/5 param=256.000000, loss=0.011390
>5/5 param=256.000000, loss=0.011595
              2         4         8        32        64       128       256
count  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000  5.000000
mean   0.018261  0.012882  0.015311  0.011961  0.012175  0.012064  0.012067
std    0.001226  0.000573  0.000798  0.000769  0.000775  0.000554  0.000736
min    0.017062  0.012065  0.014112  0.011133  0.011532  0.011488  0.011390
25%    0.017288  0.012588  0.014913  0.011369  0.011609  0.011928  0.011595
50%    0.017816  0.012946  0.015701  0.011775  0.011951  0.011940  0.011780
75%    0.019510  0.013342  0.015724  0.012704  0.012349  0.011974  0.012355
max    0.019628  0.013468  0.016104  0.012824  0.013435  0.012988  0.013215
In [43]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_learning_rate(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=0.100000, loss=0.023888
>2/5 param=0.100000, loss=0.011819
>3/5 param=0.100000, loss=0.019747
>4/5 param=0.100000, loss=0.036420
>5/5 param=0.100000, loss=0.012148
>1/5 param=0.001000, loss=0.011653
>2/5 param=0.001000, loss=0.012345
>3/5 param=0.001000, loss=0.011941
>4/5 param=0.001000, loss=0.011418
>5/5 param=0.001000, loss=0.011669
>1/5 param=0.000100, loss=0.038567
>2/5 param=0.000100, loss=0.033610
>3/5 param=0.000100, loss=0.049779
>4/5 param=0.000100, loss=0.035971
>5/5 param=0.000100, loss=0.043945
            0.1     0.001    0.0001
count  5.000000  5.000000  5.000000
mean   0.020805  0.011805  0.040375
std    0.010126  0.000354  0.006512
min    0.011819  0.011418  0.033610
25%    0.012148  0.011653  0.035971
50%    0.019747  0.011669  0.038567
75%    0.023888  0.011941  0.043945
max    0.036420  0.012345  0.049779
In [44]:
%load_ext autoreload
%autoreload 2
import tune_model
tune_model.tune_weight_regularization(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
>1/5 param=1.000000, loss=0.011683
>2/5 param=1.000000, loss=0.011673
>3/5 param=1.000000, loss=0.012200
>4/5 param=1.000000, loss=0.011844
>5/5 param=1.000000, loss=0.012050
>1/5 param=2.000000, loss=0.036623
>2/5 param=2.000000, loss=0.036176
>3/5 param=2.000000, loss=0.034430
>4/5 param=2.000000, loss=0.035556
>5/5 param=2.000000, loss=0.032201
>1/5 param=3.000000, loss=0.018865
>2/5 param=3.000000, loss=0.019171
>3/5 param=3.000000, loss=0.019481
>4/5 param=3.000000, loss=0.019422
>5/5 param=3.000000, loss=0.018394
>1/5 param=4.000000, loss=0.038356
>2/5 param=4.000000, loss=0.037684
>3/5 param=4.000000, loss=0.038813
>4/5 param=4.000000, loss=0.038209
>5/5 param=4.000000, loss=0.036820
              1         2         3         4
count  5.000000  5.000000  5.000000  5.000000
mean   0.011890  0.034997  0.019066  0.037976
std    0.000231  0.001767  0.000448  0.000762
min    0.011673  0.032201  0.018394  0.036820
25%    0.011683  0.034430  0.018865  0.037684
50%    0.011844  0.035556  0.019171  0.038209
75%    0.012050  0.036176  0.019422  0.038356
max    0.012200  0.036623  0.019481  0.038813

10. Test Improved LSTM Model

In [45]:
%load_ext autoreload
%autoreload 2
import models
model,history=models.improved_lstm_model(train_X,train_y,validation_X,validation_y)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
Train on 550 samples, validate on 53 samples
Epoch 1/500
 - 13s - loss: 0.8709 - val_loss: 0.6445
Epoch 2/500
 - 0s - loss: 0.8277 - val_loss: 0.5941
Epoch 3/500
 - 0s - loss: 0.7821 - val_loss: 0.5438
Epoch 4/500
 - 0s - loss: 0.7363 - val_loss: 0.5107
Epoch 5/500
 - 0s - loss: 0.6952 - val_loss: 0.5194
Epoch 6/500
 - 0s - loss: 0.6672 - val_loss: 0.5454
Epoch 7/500
 - 0s - loss: 0.6508 - val_loss: 0.5646
Epoch 8/500
 - 0s - loss: 0.6398 - val_loss: 0.5735
Epoch 9/500
 - 0s - loss: 0.6315 - val_loss: 0.5752
Epoch 10/500
 - 0s - loss: 0.6242 - val_loss: 0.5720
Epoch 11/500
 - 0s - loss: 0.6175 - val_loss: 0.5657
Epoch 12/500
 - 0s - loss: 0.6111 - val_loss: 0.5578
Epoch 13/500
 - 0s - loss: 0.6050 - val_loss: 0.5490
Epoch 14/500
 - 0s - loss: 0.5989 - val_loss: 0.5401
Epoch 15/500
 - 0s - loss: 0.5930 - val_loss: 0.5314
Epoch 16/500
 - 0s - loss: 0.5870 - val_loss: 0.5228
Epoch 17/500
 - 0s - loss: 0.5811 - val_loss: 0.5146
Epoch 18/500
 - 0s - loss: 0.5751 - val_loss: 0.5069
Epoch 19/500
 - 0s - loss: 0.5691 - val_loss: 0.4998
Epoch 20/500
 - 0s - loss: 0.5631 - val_loss: 0.4931
Epoch 21/500
 - 0s - loss: 0.5570 - val_loss: 0.4867
Epoch 22/500
 - 0s - loss: 0.5508 - val_loss: 0.4805
Epoch 23/500
 - 0s - loss: 0.5447 - val_loss: 0.4744
Epoch 24/500
 - 0s - loss: 0.5385 - val_loss: 0.4683
Epoch 25/500
 - 0s - loss: 0.5323 - val_loss: 0.4621
Epoch 26/500
 - 0s - loss: 0.5261 - val_loss: 0.4560
Epoch 27/500
 - 0s - loss: 0.5199 - val_loss: 0.4498
Epoch 28/500
 - 0s - loss: 0.5137 - val_loss: 0.4436
Epoch 29/500
 - 0s - loss: 0.5075 - val_loss: 0.4374
Epoch 30/500
 - 0s - loss: 0.5013 - val_loss: 0.4314
Epoch 31/500
 - 0s - loss: 0.4950 - val_loss: 0.4256
Epoch 32/500
 - 0s - loss: 0.4887 - val_loss: 0.4198
Epoch 33/500
 - 0s - loss: 0.4823 - val_loss: 0.4140
Epoch 34/500
 - 0s - loss: 0.4760 - val_loss: 0.4082
Epoch 35/500
 - 0s - loss: 0.4695 - val_loss: 0.4024
Epoch 36/500
 - 0s - loss: 0.4631 - val_loss: 0.3967
Epoch 37/500
 - 0s - loss: 0.4566 - val_loss: 0.3909
Epoch 38/500
 - 0s - loss: 0.4501 - val_loss: 0.3850
Epoch 39/500
 - 0s - loss: 0.4435 - val_loss: 0.3790
Epoch 40/500
 - 0s - loss: 0.4370 - val_loss: 0.3728
Epoch 41/500
 - 0s - loss: 0.4305 - val_loss: 0.3665
Epoch 42/500
 - 0s - loss: 0.4239 - val_loss: 0.3604
Epoch 43/500
 - 0s - loss: 0.4173 - val_loss: 0.3545
Epoch 44/500
 - 0s - loss: 0.4106 - val_loss: 0.3494
Epoch 45/500
 - 0s - loss: 0.4037 - val_loss: 0.3445
Epoch 46/500
 - 0s - loss: 0.3968 - val_loss: 0.3397
Epoch 47/500
 - 0s - loss: 0.3899 - val_loss: 0.3344
Epoch 48/500
 - 0s - loss: 0.3831 - val_loss: 0.3291
Epoch 49/500
 - 0s - loss: 0.3762 - val_loss: 0.3238
Epoch 50/500
 - 0s - loss: 0.3695 - val_loss: 0.3186
Epoch 51/500
 - 0s - loss: 0.3628 - val_loss: 0.3132
Epoch 52/500
 - 0s - loss: 0.3564 - val_loss: 0.3083
Epoch 53/500
 - 0s - loss: 0.3501 - val_loss: 0.3039
Epoch 54/500
 - 0s - loss: 0.3439 - val_loss: 0.2999
Epoch 55/500
 - 0s - loss: 0.3377 - val_loss: 0.2965
Epoch 56/500
 - 0s - loss: 0.3316 - val_loss: 0.2929
Epoch 57/500
 - 0s - loss: 0.3259 - val_loss: 0.2895
Epoch 58/500
 - 0s - loss: 0.3204 - val_loss: 0.2862
Epoch 59/500
 - 0s - loss: 0.3151 - val_loss: 0.2832
Epoch 60/500
 - 0s - loss: 0.3099 - val_loss: 0.2804
Epoch 61/500
 - 0s - loss: 0.3048 - val_loss: 0.2777
Epoch 62/500
 - 0s - loss: 0.2997 - val_loss: 0.2747
Epoch 63/500
 - 0s - loss: 0.2948 - val_loss: 0.2717
Epoch 64/500
 - 0s - loss: 0.2903 - val_loss: 0.2692
Epoch 65/500
 - 0s - loss: 0.2860 - val_loss: 0.2670
Epoch 66/500
 - 0s - loss: 0.2816 - val_loss: 0.2640
Epoch 67/500
 - 0s - loss: 0.2775 - val_loss: 0.2608
Epoch 68/500
 - 0s - loss: 0.2736 - val_loss: 0.2577
Epoch 69/500
 - 0s - loss: 0.2698 - val_loss: 0.2544
Epoch 70/500
 - 0s - loss: 0.2662 - val_loss: 0.2512
Epoch 71/500
 - 0s - loss: 0.2627 - val_loss: 0.2481
Epoch 72/500
 - 0s - loss: 0.2592 - val_loss: 0.2450
Epoch 73/500
 - 0s - loss: 0.2559 - val_loss: 0.2416
Epoch 74/500
 - 0s - loss: 0.2527 - val_loss: 0.2384
Epoch 75/500
 - 0s - loss: 0.2496 - val_loss: 0.2357
Epoch 76/500
 - 0s - loss: 0.2465 - val_loss: 0.2329
Epoch 77/500
 - 0s - loss: 0.2434 - val_loss: 0.2297
Epoch 78/500
 - 0s - loss: 0.2405 - val_loss: 0.2269
Epoch 79/500
 - 0s - loss: 0.2376 - val_loss: 0.2240
Epoch 80/500
 - 0s - loss: 0.2347 - val_loss: 0.2208
Epoch 81/500
 - 0s - loss: 0.2320 - val_loss: 0.2179
Epoch 82/500
 - 0s - loss: 0.2292 - val_loss: 0.2152
Epoch 83/500
 - 0s - loss: 0.2266 - val_loss: 0.2124
Epoch 84/500
 - 0s - loss: 0.2239 - val_loss: 0.2094
Epoch 85/500
 - 0s - loss: 0.2213 - val_loss: 0.2067
Epoch 86/500
 - 0s - loss: 0.2188 - val_loss: 0.2042
Epoch 87/500
 - 0s - loss: 0.2162 - val_loss: 0.2014
Epoch 88/500
 - 0s - loss: 0.2138 - val_loss: 0.1988
Epoch 89/500
 - 0s - loss: 0.2113 - val_loss: 0.1963
Epoch 90/500
 - 0s - loss: 0.2089 - val_loss: 0.1937
Epoch 91/500
 - 0s - loss: 0.2065 - val_loss: 0.1911
Epoch 92/500
 - 0s - loss: 0.2041 - val_loss: 0.1889
Epoch 93/500
 - 0s - loss: 0.2018 - val_loss: 0.1864
Epoch 94/500
 - 0s - loss: 0.1995 - val_loss: 0.1837
Epoch 95/500
 - 0s - loss: 0.1972 - val_loss: 0.1813
Epoch 96/500
 - 0s - loss: 0.1949 - val_loss: 0.1792
Epoch 97/500
 - 0s - loss: 0.1927 - val_loss: 0.1767
Epoch 98/500
 - 0s - loss: 0.1905 - val_loss: 0.1744
Epoch 99/500
 - 0s - loss: 0.1883 - val_loss: 0.1723
Epoch 100/500
 - 0s - loss: 0.1862 - val_loss: 0.1700
Epoch 101/500
 - 0s - loss: 0.1841 - val_loss: 0.1678
Epoch 102/500
 - 0s - loss: 0.1820 - val_loss: 0.1657
Epoch 103/500
 - 0s - loss: 0.1799 - val_loss: 0.1636
Epoch 104/500
 - 0s - loss: 0.1778 - val_loss: 0.1614
Epoch 105/500
 - 0s - loss: 0.1758 - val_loss: 0.1594
Epoch 106/500
 - 0s - loss: 0.1738 - val_loss: 0.1574
Epoch 107/500
 - 0s - loss: 0.1718 - val_loss: 0.1553
Epoch 108/500
 - 0s - loss: 0.1699 - val_loss: 0.1534
Epoch 109/500
 - 0s - loss: 0.1679 - val_loss: 0.1515
Epoch 110/500
 - 0s - loss: 0.1660 - val_loss: 0.1494
Epoch 111/500
 - 0s - loss: 0.1641 - val_loss: 0.1474
Epoch 112/500
 - 0s - loss: 0.1622 - val_loss: 0.1456
Epoch 113/500
 - 0s - loss: 0.1604 - val_loss: 0.1438
Epoch 114/500
 - 0s - loss: 0.1585 - val_loss: 0.1419
Epoch 115/500
 - 0s - loss: 0.1567 - val_loss: 0.1400
Epoch 116/500
 - 0s - loss: 0.1549 - val_loss: 0.1383
Epoch 117/500
 - 0s - loss: 0.1531 - val_loss: 0.1365
Epoch 118/500
 - 0s - loss: 0.1514 - val_loss: 0.1347
Epoch 119/500
 - 0s - loss: 0.1496 - val_loss: 0.1330
Epoch 120/500
 - 0s - loss: 0.1479 - val_loss: 0.1313
Epoch 121/500
 - 0s - loss: 0.1462 - val_loss: 0.1296
Epoch 122/500
 - 0s - loss: 0.1445 - val_loss: 0.1279
Epoch 123/500
 - 0s - loss: 0.1429 - val_loss: 0.1262
Epoch 124/500
 - 0s - loss: 0.1412 - val_loss: 0.1245
Epoch 125/500
 - 0s - loss: 0.1396 - val_loss: 0.1229
Epoch 126/500
 - 0s - loss: 0.1380 - val_loss: 0.1213
Epoch 127/500
 - 0s - loss: 0.1364 - val_loss: 0.1197
Epoch 128/500
 - 0s - loss: 0.1348 - val_loss: 0.1181
Epoch 129/500
 - 0s - loss: 0.1333 - val_loss: 0.1166
Epoch 130/500
 - 0s - loss: 0.1317 - val_loss: 0.1150
Epoch 131/500
 - 0s - loss: 0.1302 - val_loss: 0.1135
Epoch 132/500
 - 0s - loss: 0.1287 - val_loss: 0.1120
Epoch 133/500
 - 0s - loss: 0.1272 - val_loss: 0.1106
Epoch 134/500
 - 0s - loss: 0.1258 - val_loss: 0.1091
Epoch 135/500
 - 0s - loss: 0.1243 - val_loss: 0.1077
Epoch 136/500
 - 0s - loss: 0.1229 - val_loss: 0.1062
Epoch 137/500
 - 0s - loss: 0.1215 - val_loss: 0.1048
Epoch 138/500
 - 0s - loss: 0.1201 - val_loss: 0.1035
Epoch 139/500
 - 0s - loss: 0.1187 - val_loss: 0.1022
Epoch 140/500
 - 0s - loss: 0.1173 - val_loss: 0.1007
Epoch 141/500
 - 0s - loss: 0.1160 - val_loss: 0.0995
Epoch 142/500
 - 0s - loss: 0.1146 - val_loss: 0.0981
Epoch 143/500
 - 0s - loss: 0.1133 - val_loss: 0.0967
Epoch 144/500
 - 0s - loss: 0.1120 - val_loss: 0.0955
Epoch 145/500
 - 0s - loss: 0.1107 - val_loss: 0.0942
Epoch 146/500
 - 0s - loss: 0.1095 - val_loss: 0.0930
Epoch 147/500
 - 0s - loss: 0.1082 - val_loss: 0.0918
Epoch 148/500
 - 0s - loss: 0.1070 - val_loss: 0.0905
Epoch 149/500
 - 0s - loss: 0.1057 - val_loss: 0.0894
Epoch 150/500
 - 0s - loss: 0.1045 - val_loss: 0.0881
Epoch 151/500
 - 0s - loss: 0.1033 - val_loss: 0.0870
Epoch 152/500
 - 0s - loss: 0.1021 - val_loss: 0.0857
Epoch 153/500
 - 0s - loss: 0.1010 - val_loss: 0.0847
Epoch 154/500
 - 0s - loss: 0.0998 - val_loss: 0.0835
Epoch 155/500
 - 0s - loss: 0.0987 - val_loss: 0.0824
Epoch 156/500
 - 0s - loss: 0.0975 - val_loss: 0.0814
Epoch 157/500
 - 0s - loss: 0.0964 - val_loss: 0.0801
Epoch 158/500
 - 0s - loss: 0.0953 - val_loss: 0.0792
Epoch 159/500
 - 0s - loss: 0.0943 - val_loss: 0.0782
Epoch 160/500
 - 0s - loss: 0.0932 - val_loss: 0.0769
Epoch 161/500
 - 0s - loss: 0.0921 - val_loss: 0.0761
Epoch 162/500
 - 0s - loss: 0.0911 - val_loss: 0.0751
Epoch 163/500
 - 0s - loss: 0.0901 - val_loss: 0.0739
Epoch 164/500
 - 0s - loss: 0.0890 - val_loss: 0.0730
Epoch 165/500
 - 0s - loss: 0.0880 - val_loss: 0.0720
Epoch 166/500
 - 0s - loss: 0.0870 - val_loss: 0.0711
Epoch 167/500
 - 0s - loss: 0.0861 - val_loss: 0.0701
Epoch 168/500
 - 0s - loss: 0.0851 - val_loss: 0.0692
Epoch 169/500
 - 0s - loss: 0.0842 - val_loss: 0.0681
Epoch 170/500
 - 0s - loss: 0.0832 - val_loss: 0.0674
Epoch 171/500
 - 0s - loss: 0.0823 - val_loss: 0.0665
Epoch 172/500
 - 0s - loss: 0.0814 - val_loss: 0.0655
Epoch 173/500
 - 0s - loss: 0.0804 - val_loss: 0.0646
Epoch 174/500
 - 0s - loss: 0.0796 - val_loss: 0.0638
Epoch 175/500
 - 0s - loss: 0.0787 - val_loss: 0.0628
Epoch 176/500
 - 0s - loss: 0.0778 - val_loss: 0.0621
Epoch 177/500
 - 0s - loss: 0.0770 - val_loss: 0.0613
Epoch 178/500
 - 0s - loss: 0.0761 - val_loss: 0.0604
Epoch 179/500
 - 0s - loss: 0.0753 - val_loss: 0.0596
Epoch 180/500
 - 0s - loss: 0.0744 - val_loss: 0.0588
Epoch 181/500
 - 0s - loss: 0.0736 - val_loss: 0.0580
Epoch 182/500
 - 0s - loss: 0.0728 - val_loss: 0.0573
Epoch 183/500
 - 0s - loss: 0.0720 - val_loss: 0.0565
Epoch 184/500
 - 0s - loss: 0.0713 - val_loss: 0.0557
Epoch 185/500
 - 0s - loss: 0.0705 - val_loss: 0.0550
Epoch 186/500
 - 0s - loss: 0.0697 - val_loss: 0.0542
Epoch 187/500
 - 0s - loss: 0.0690 - val_loss: 0.0535
Epoch 188/500
 - 0s - loss: 0.0683 - val_loss: 0.0528
Epoch 189/500
 - 0s - loss: 0.0675 - val_loss: 0.0521
Epoch 190/500
 - 0s - loss: 0.0668 - val_loss: 0.0514
Epoch 191/500
 - 0s - loss: 0.0661 - val_loss: 0.0508
Epoch 192/500
 - 0s - loss: 0.0654 - val_loss: 0.0500
Epoch 193/500
 - 0s - loss: 0.0647 - val_loss: 0.0494
Epoch 194/500
 - 0s - loss: 0.0641 - val_loss: 0.0488
Epoch 195/500
 - 0s - loss: 0.0634 - val_loss: 0.0481
Epoch 196/500
 - 0s - loss: 0.0627 - val_loss: 0.0475
Epoch 197/500
 - 0s - loss: 0.0621 - val_loss: 0.0468
Epoch 198/500
 - 0s - loss: 0.0615 - val_loss: 0.0462
Epoch 199/500
 - 0s - loss: 0.0608 - val_loss: 0.0456
Epoch 200/500
 - 0s - loss: 0.0602 - val_loss: 0.0451
Epoch 201/500
 - 0s - loss: 0.0596 - val_loss: 0.0444
Epoch 202/500
 - 0s - loss: 0.0590 - val_loss: 0.0438
Epoch 203/500
 - 0s - loss: 0.0584 - val_loss: 0.0434
Epoch 204/500
 - 0s - loss: 0.0578 - val_loss: 0.0426
Epoch 205/500
 - 0s - loss: 0.0573 - val_loss: 0.0420
Epoch 206/500
 - 0s - loss: 0.0567 - val_loss: 0.0419
Epoch 207/500
 - 0s - loss: 0.0561 - val_loss: 0.0412
Epoch 208/500
 - 0s - loss: 0.0556 - val_loss: 0.0404
Epoch 209/500
 - 0s - loss: 0.0550 - val_loss: 0.0401
Epoch 210/500
 - 0s - loss: 0.0545 - val_loss: 0.0397
Epoch 211/500
 - 0s - loss: 0.0540 - val_loss: 0.0390
Epoch 212/500
 - 0s - loss: 0.0535 - val_loss: 0.0384
Epoch 213/500
 - 0s - loss: 0.0529 - val_loss: 0.0382
Epoch 214/500
 - 0s - loss: 0.0525 - val_loss: 0.0375
Epoch 215/500
 - 0s - loss: 0.0520 - val_loss: 0.0371
Epoch 216/500
 - 0s - loss: 0.0515 - val_loss: 0.0367
Epoch 217/500
 - 0s - loss: 0.0510 - val_loss: 0.0361
Epoch 218/500
 - 0s - loss: 0.0505 - val_loss: 0.0357
Epoch 219/500
 - 0s - loss: 0.0500 - val_loss: 0.0354
Epoch 220/500
 - 0s - loss: 0.0496 - val_loss: 0.0348
Epoch 221/500
 - 0s - loss: 0.0491 - val_loss: 0.0343
Epoch 222/500
 - 0s - loss: 0.0487 - val_loss: 0.0339
Epoch 223/500
 - 0s - loss: 0.0483 - val_loss: 0.0335
Epoch 224/500
 - 0s - loss: 0.0478 - val_loss: 0.0330
Epoch 225/500
 - 0s - loss: 0.0474 - val_loss: 0.0328
Epoch 226/500
 - 0s - loss: 0.0470 - val_loss: 0.0322
Epoch 227/500
 - 0s - loss: 0.0466 - val_loss: 0.0319
Epoch 228/500
 - 0s - loss: 0.0462 - val_loss: 0.0316
Epoch 229/500
 - 0s - loss: 0.0458 - val_loss: 0.0312
Epoch 230/500
 - 0s - loss: 0.0454 - val_loss: 0.0308
Epoch 231/500
 - 0s - loss: 0.0450 - val_loss: 0.0304
Epoch 232/500
 - 0s - loss: 0.0446 - val_loss: 0.0301
Epoch 233/500
 - 0s - loss: 0.0443 - val_loss: 0.0298
Epoch 234/500
 - 0s - loss: 0.0439 - val_loss: 0.0293
Epoch 235/500
 - 0s - loss: 0.0435 - val_loss: 0.0293
Epoch 236/500
 - 0s - loss: 0.0432 - val_loss: 0.0285
Epoch 237/500
 - 0s - loss: 0.0429 - val_loss: 0.0287
Epoch 238/500
 - 0s - loss: 0.0425 - val_loss: 0.0280
Epoch 239/500
 - 0s - loss: 0.0422 - val_loss: 0.0279
Epoch 240/500
 - 0s - loss: 0.0419 - val_loss: 0.0273
Epoch 241/500
 - 0s - loss: 0.0415 - val_loss: 0.0274
Epoch 242/500
 - 0s - loss: 0.0412 - val_loss: 0.0268
Epoch 243/500
 - 0s - loss: 0.0409 - val_loss: 0.0267
Epoch 244/500
 - 0s - loss: 0.0406 - val_loss: 0.0262
Epoch 245/500
 - 0s - loss: 0.0403 - val_loss: 0.0262
Epoch 246/500
 - 0s - loss: 0.0400 - val_loss: 0.0257
Epoch 247/500
 - 0s - loss: 0.0397 - val_loss: 0.0257
Epoch 248/500
 - 0s - loss: 0.0394 - val_loss: 0.0249
Epoch 249/500
 - 0s - loss: 0.0391 - val_loss: 0.0251
Epoch 250/500
 - 0s - loss: 0.0389 - val_loss: 0.0245
Epoch 251/500
 - 0s - loss: 0.0386 - val_loss: 0.0245
Epoch 252/500
 - 0s - loss: 0.0383 - val_loss: 0.0240
Epoch 253/500
 - 0s - loss: 0.0381 - val_loss: 0.0240
Epoch 254/500
 - 0s - loss: 0.0378 - val_loss: 0.0236
Epoch 255/500
 - 0s - loss: 0.0376 - val_loss: 0.0235
Epoch 256/500
 - 0s - loss: 0.0373 - val_loss: 0.0231
Epoch 257/500
 - 0s - loss: 0.0371 - val_loss: 0.0231
Epoch 258/500
 - 0s - loss: 0.0368 - val_loss: 0.0226
Epoch 259/500
 - 0s - loss: 0.0366 - val_loss: 0.0225
Epoch 260/500
 - 0s - loss: 0.0364 - val_loss: 0.0221
Epoch 261/500
 - 0s - loss: 0.0361 - val_loss: 0.0221
Epoch 262/500
 - 0s - loss: 0.0359 - val_loss: 0.0217
Epoch 263/500
 - 0s - loss: 0.0357 - val_loss: 0.0217
Epoch 264/500
 - 0s - loss: 0.0355 - val_loss: 0.0213
Epoch 265/500
 - 0s - loss: 0.0353 - val_loss: 0.0213
Epoch 266/500
 - 0s - loss: 0.0351 - val_loss: 0.0209
Epoch 267/500
 - 0s - loss: 0.0349 - val_loss: 0.0209
Epoch 268/500
 - 0s - loss: 0.0347 - val_loss: 0.0206
Epoch 269/500
 - 0s - loss: 0.0345 - val_loss: 0.0205
Epoch 270/500
 - 0s - loss: 0.0343 - val_loss: 0.0203
Epoch 271/500
 - 0s - loss: 0.0341 - val_loss: 0.0202
Epoch 272/500
 - 0s - loss: 0.0339 - val_loss: 0.0199
Epoch 273/500
 - 0s - loss: 0.0337 - val_loss: 0.0198
Epoch 274/500
 - 0s - loss: 0.0335 - val_loss: 0.0196
Epoch 275/500
 - 0s - loss: 0.0334 - val_loss: 0.0195
Epoch 276/500
 - 0s - loss: 0.0332 - val_loss: 0.0192
Epoch 277/500
 - 0s - loss: 0.0330 - val_loss: 0.0192
Epoch 278/500
 - 0s - loss: 0.0329 - val_loss: 0.0190
Epoch 279/500
 - 0s - loss: 0.0327 - val_loss: 0.0188
Epoch 280/500
 - 0s - loss: 0.0326 - val_loss: 0.0186
Epoch 281/500
 - 0s - loss: 0.0324 - val_loss: 0.0185
Epoch 282/500
 - 0s - loss: 0.0322 - val_loss: 0.0184
Epoch 283/500
 - 0s - loss: 0.0321 - val_loss: 0.0181
Epoch 284/500
 - 0s - loss: 0.0319 - val_loss: 0.0182
Epoch 285/500
 - 0s - loss: 0.0318 - val_loss: 0.0179
Epoch 286/500
 - 0s - loss: 0.0317 - val_loss: 0.0178
Epoch 287/500
 - 0s - loss: 0.0315 - val_loss: 0.0176
Epoch 288/500
 - 0s - loss: 0.0314 - val_loss: 0.0175
Epoch 289/500
 - 0s - loss: 0.0313 - val_loss: 0.0173
Epoch 290/500
 - 0s - loss: 0.0311 - val_loss: 0.0173
Epoch 291/500
 - 0s - loss: 0.0310 - val_loss: 0.0171
Epoch 292/500
 - 0s - loss: 0.0309 - val_loss: 0.0170
Epoch 293/500
 - 0s - loss: 0.0308 - val_loss: 0.0169
Epoch 294/500
 - 0s - loss: 0.0306 - val_loss: 0.0168
Epoch 295/500
 - 0s - loss: 0.0305 - val_loss: 0.0167
Epoch 296/500
 - 0s - loss: 0.0304 - val_loss: 0.0166
Epoch 297/500
 - 0s - loss: 0.0303 - val_loss: 0.0165
Epoch 298/500
 - 0s - loss: 0.0302 - val_loss: 0.0164
Epoch 299/500
 - 0s - loss: 0.0301 - val_loss: 0.0162
Epoch 300/500
 - 0s - loss: 0.0300 - val_loss: 0.0162
Epoch 301/500
 - 0s - loss: 0.0299 - val_loss: 0.0161
Epoch 302/500
 - 0s - loss: 0.0298 - val_loss: 0.0160
Epoch 303/500
 - 0s - loss: 0.0297 - val_loss: 0.0159
Epoch 304/500
 - 0s - loss: 0.0296 - val_loss: 0.0159
Epoch 305/500
 - 0s - loss: 0.0295 - val_loss: 0.0158
Epoch 306/500
 - 0s - loss: 0.0294 - val_loss: 0.0157
Epoch 307/500
 - 0s - loss: 0.0293 - val_loss: 0.0156
Epoch 308/500
 - 0s - loss: 0.0292 - val_loss: 0.0155
Epoch 309/500
 - 0s - loss: 0.0291 - val_loss: 0.0154
Epoch 310/500
 - 0s - loss: 0.0290 - val_loss: 0.0154
Epoch 311/500
 - 0s - loss: 0.0290 - val_loss: 0.0153
Epoch 312/500
 - 0s - loss: 0.0289 - val_loss: 0.0152
Epoch 313/500
 - 0s - loss: 0.0288 - val_loss: 0.0151
Epoch 314/500
 - 0s - loss: 0.0287 - val_loss: 0.0150
Epoch 315/500
 - 0s - loss: 0.0286 - val_loss: 0.0150
Epoch 316/500
 - 0s - loss: 0.0286 - val_loss: 0.0148
Epoch 317/500
 - 0s - loss: 0.0285 - val_loss: 0.0149
Epoch 318/500
 - 0s - loss: 0.0284 - val_loss: 0.0147
Epoch 319/500
 - 0s - loss: 0.0283 - val_loss: 0.0148
Epoch 320/500
 - 0s - loss: 0.0283 - val_loss: 0.0148
Epoch 321/500
 - 0s - loss: 0.0282 - val_loss: 0.0146
Epoch 322/500
 - 0s - loss: 0.0282 - val_loss: 0.0145
Epoch 323/500
 - 0s - loss: 0.0281 - val_loss: 0.0144
Epoch 324/500
 - 0s - loss: 0.0280 - val_loss: 0.0145
Epoch 325/500
 - 0s - loss: 0.0279 - val_loss: 0.0144
Epoch 326/500
 - 0s - loss: 0.0279 - val_loss: 0.0143
Epoch 327/500
 - 0s - loss: 0.0278 - val_loss: 0.0142
Epoch 328/500
 - 0s - loss: 0.0278 - val_loss: 0.0143
Epoch 329/500
 - 0s - loss: 0.0277 - val_loss: 0.0142
Epoch 330/500
 - 0s - loss: 0.0277 - val_loss: 0.0144
Epoch 331/500
 - 0s - loss: 0.0276 - val_loss: 0.0140
Epoch 332/500
 - 0s - loss: 0.0275 - val_loss: 0.0142
Epoch 333/500
 - 0s - loss: 0.0275 - val_loss: 0.0139
Epoch 334/500
 - 0s - loss: 0.0275 - val_loss: 0.0140
Epoch 335/500
 - 0s - loss: 0.0274 - val_loss: 0.0140
Epoch 336/500
 - 0s - loss: 0.0273 - val_loss: 0.0141
Epoch 337/500
 - 0s - loss: 0.0273 - val_loss: 0.0138
Epoch 338/500
 - 0s - loss: 0.0273 - val_loss: 0.0140
Epoch 339/500
 - 0s - loss: 0.0272 - val_loss: 0.0136
Epoch 340/500
 - 0s - loss: 0.0272 - val_loss: 0.0139
Epoch 341/500
 - 0s - loss: 0.0271 - val_loss: 0.0136
Epoch 342/500
 - 0s - loss: 0.0271 - val_loss: 0.0140
Epoch 343/500
 - 0s - loss: 0.0271 - val_loss: 0.0134
Epoch 344/500
 - 0s - loss: 0.0270 - val_loss: 0.0137
Epoch 345/500
 - 0s - loss: 0.0270 - val_loss: 0.0135
Epoch 346/500
 - 0s - loss: 0.0269 - val_loss: 0.0139
Epoch 347/500
 - 0s - loss: 0.0269 - val_loss: 0.0134
Epoch 348/500
 - 0s - loss: 0.0269 - val_loss: 0.0139
Epoch 349/500
 - 0s - loss: 0.0269 - val_loss: 0.0133
Epoch 350/500
 - 0s - loss: 0.0268 - val_loss: 0.0135
Epoch 351/500
 - 0s - loss: 0.0268 - val_loss: 0.0134
Epoch 352/500
 - 0s - loss: 0.0267 - val_loss: 0.0136
Epoch 353/500
 - 0s - loss: 0.0267 - val_loss: 0.0131
Epoch 354/500
 - 0s - loss: 0.0267 - val_loss: 0.0135
Epoch 355/500
 - 0s - loss: 0.0267 - val_loss: 0.0131
Epoch 356/500
 - 0s - loss: 0.0266 - val_loss: 0.0138
Epoch 357/500
 - 0s - loss: 0.0267 - val_loss: 0.0129
Epoch 358/500
 - 0s - loss: 0.0266 - val_loss: 0.0137
Epoch 359/500
 - 0s - loss: 0.0266 - val_loss: 0.0130
Epoch 360/500
 - 0s - loss: 0.0266 - val_loss: 0.0136
Epoch 361/500
 - 0s - loss: 0.0266 - val_loss: 0.0130
Epoch 362/500
 - 0s - loss: 0.0267 - val_loss: 0.0133
Epoch 363/500
 - 0s - loss: 0.0266 - val_loss: 0.0129
Epoch 364/500
 - 0s - loss: 0.0266 - val_loss: 0.0129
Epoch 365/500
 - 0s - loss: 0.0266 - val_loss: 0.0128
Epoch 366/500
 - 0s - loss: 0.0266 - val_loss: 0.0127
Epoch 367/500
 - 0s - loss: 0.0265 - val_loss: 0.0127
Epoch 368/500
 - 0s - loss: 0.0265 - val_loss: 0.0125
Epoch 369/500
 - 0s - loss: 0.0264 - val_loss: 0.0126
Epoch 370/500
 - 0s - loss: 0.0264 - val_loss: 0.0125
Epoch 371/500
 - 0s - loss: 0.0264 - val_loss: 0.0126
Epoch 372/500
 - 0s - loss: 0.0263 - val_loss: 0.0125
Epoch 373/500
 - 0s - loss: 0.0262 - val_loss: 0.0126
Epoch 374/500
 - 0s - loss: 0.0262 - val_loss: 0.0126
Epoch 375/500
 - 0s - loss: 0.0262 - val_loss: 0.0127
Epoch 376/500
 - 0s - loss: 0.0262 - val_loss: 0.0127
Epoch 377/500
 - 0s - loss: 0.0261 - val_loss: 0.0128
Epoch 378/500
 - 0s - loss: 0.0261 - val_loss: 0.0127
Epoch 379/500
 - 0s - loss: 0.0261 - val_loss: 0.0129
Epoch 380/500
 - 0s - loss: 0.0261 - val_loss: 0.0128
Epoch 381/500
 - 0s - loss: 0.0261 - val_loss: 0.0130
Epoch 382/500
 - 0s - loss: 0.0260 - val_loss: 0.0126
Epoch 383/500
 - 0s - loss: 0.0260 - val_loss: 0.0128
Epoch 384/500
 - 0s - loss: 0.0260 - val_loss: 0.0127
Epoch 385/500
 - 0s - loss: 0.0260 - val_loss: 0.0129
Epoch 386/500
 - 0s - loss: 0.0260 - val_loss: 0.0127
Epoch 387/500
 - 0s - loss: 0.0259 - val_loss: 0.0127
Epoch 388/500
 - 0s - loss: 0.0260 - val_loss: 0.0127
Epoch 389/500
 - 0s - loss: 0.0260 - val_loss: 0.0126
Epoch 390/500
 - 0s - loss: 0.0259 - val_loss: 0.0128
Epoch 391/500
 - 0s - loss: 0.0259 - val_loss: 0.0125
Epoch 392/500
 - 0s - loss: 0.0259 - val_loss: 0.0129
Epoch 393/500
 - 0s - loss: 0.0259 - val_loss: 0.0123
Epoch 394/500
 - 0s - loss: 0.0259 - val_loss: 0.0127
Epoch 395/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 396/500
 - 0s - loss: 0.0259 - val_loss: 0.0130
Epoch 397/500
 - 0s - loss: 0.0259 - val_loss: 0.0123
Epoch 398/500
 - 0s - loss: 0.0258 - val_loss: 0.0127
Epoch 399/500
 - 0s - loss: 0.0259 - val_loss: 0.0124
Epoch 400/500
 - 0s - loss: 0.0258 - val_loss: 0.0129
Epoch 401/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 402/500
 - 0s - loss: 0.0258 - val_loss: 0.0126
Epoch 403/500
 - 0s - loss: 0.0258 - val_loss: 0.0123
Epoch 404/500
 - 0s - loss: 0.0258 - val_loss: 0.0127
Epoch 405/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 406/500
 - 0s - loss: 0.0258 - val_loss: 0.0125
Epoch 407/500
 - 0s - loss: 0.0257 - val_loss: 0.0123
Epoch 408/500
 - 0s - loss: 0.0258 - val_loss: 0.0127
Epoch 409/500
 - 0s - loss: 0.0258 - val_loss: 0.0122
Epoch 410/500
 - 0s - loss: 0.0257 - val_loss: 0.0126
Epoch 411/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 412/500
 - 0s - loss: 0.0257 - val_loss: 0.0127
Epoch 413/500
 - 0s - loss: 0.0258 - val_loss: 0.0121
Epoch 414/500
 - 0s - loss: 0.0257 - val_loss: 0.0125
Epoch 415/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 416/500
 - 0s - loss: 0.0257 - val_loss: 0.0126
Epoch 417/500
 - 0s - loss: 0.0257 - val_loss: 0.0123
Epoch 418/500
 - 0s - loss: 0.0257 - val_loss: 0.0124
Epoch 419/500
 - 0s - loss: 0.0257 - val_loss: 0.0122
Epoch 420/500
 - 0s - loss: 0.0257 - val_loss: 0.0124
Epoch 421/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 422/500
 - 0s - loss: 0.0257 - val_loss: 0.0126
Epoch 423/500
 - 0s - loss: 0.0257 - val_loss: 0.0121
Epoch 424/500
 - 0s - loss: 0.0257 - val_loss: 0.0126
Epoch 425/500
 - 0s - loss: 0.0256 - val_loss: 0.0122
Epoch 426/500
 - 0s - loss: 0.0257 - val_loss: 0.0127
Epoch 427/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 428/500
 - 0s - loss: 0.0257 - val_loss: 0.0125
Epoch 429/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 430/500
 - 0s - loss: 0.0257 - val_loss: 0.0126
Epoch 431/500
 - 0s - loss: 0.0257 - val_loss: 0.0120
Epoch 432/500
 - 0s - loss: 0.0257 - val_loss: 0.0127
Epoch 433/500
 - 0s - loss: 0.0257 - val_loss: 0.0119
Epoch 434/500
 - 0s - loss: 0.0256 - val_loss: 0.0125
Epoch 435/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 436/500
 - 0s - loss: 0.0256 - val_loss: 0.0123
Epoch 437/500
 - 0s - loss: 0.0256 - val_loss: 0.0120
Epoch 438/500
 - 0s - loss: 0.0256 - val_loss: 0.0124
Epoch 439/500
 - 0s - loss: 0.0256 - val_loss: 0.0121
Epoch 440/500
 - 0s - loss: 0.0256 - val_loss: 0.0124
Epoch 441/500
 - 0s - loss: 0.0256 - val_loss: 0.0119
Epoch 442/500
 - 0s - loss: 0.0256 - val_loss: 0.0127
Epoch 443/500
 - 0s - loss: 0.0257 - val_loss: 0.0119
Epoch 444/500
 - 0s - loss: 0.0255 - val_loss: 0.0126
Epoch 445/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 446/500
 - 0s - loss: 0.0256 - val_loss: 0.0123
Epoch 447/500
 - 0s - loss: 0.0256 - val_loss: 0.0119
Epoch 448/500
 - 0s - loss: 0.0255 - val_loss: 0.0123
Epoch 449/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 450/500
 - 0s - loss: 0.0256 - val_loss: 0.0123
Epoch 451/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 452/500
 - 0s - loss: 0.0255 - val_loss: 0.0125
Epoch 453/500
 - 0s - loss: 0.0255 - val_loss: 0.0121
Epoch 454/500
 - 0s - loss: 0.0255 - val_loss: 0.0123
Epoch 455/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 456/500
 - 0s - loss: 0.0255 - val_loss: 0.0123
Epoch 457/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 458/500
 - 0s - loss: 0.0255 - val_loss: 0.0123
Epoch 459/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 460/500
 - 0s - loss: 0.0255 - val_loss: 0.0123
Epoch 461/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 462/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 463/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 464/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 465/500
 - 0s - loss: 0.0255 - val_loss: 0.0120
Epoch 466/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 467/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 468/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 469/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 470/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 471/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 472/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 473/500
 - 0s - loss: 0.0254 - val_loss: 0.0123
Epoch 474/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 475/500
 - 0s - loss: 0.0254 - val_loss: 0.0123
Epoch 476/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 477/500
 - 0s - loss: 0.0254 - val_loss: 0.0119
Epoch 478/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 479/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 480/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 481/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 482/500
 - 0s - loss: 0.0254 - val_loss: 0.0123
Epoch 483/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 484/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 485/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 486/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 487/500
 - 0s - loss: 0.0254 - val_loss: 0.0123
Epoch 488/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 489/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 490/500
 - 0s - loss: 0.0254 - val_loss: 0.0123
Epoch 491/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 492/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 493/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 494/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 495/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 496/500
 - 0s - loss: 0.0254 - val_loss: 0.0121
Epoch 497/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
Epoch 498/500
 - 0s - loss: 0.0253 - val_loss: 0.0120
Epoch 499/500
 - 0s - loss: 0.0254 - val_loss: 0.0122
Epoch 500/500
 - 0s - loss: 0.0254 - val_loss: 0.0120
In [46]:
pyplot.plot(history['loss'], label='train')
pyplot.plot(history['val_loss'], label='validation')
pyplot.legend()
pyplot.show()

Test improved model on Validation Data

In [47]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(validation_X,validation_y,model,scaler)
print('LSTM Model on Validation Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Model on Validation Data RMSE: 8.958

Test improved model on Unseen(Test) Data

In [48]:
# make a prediction
%load_ext autoreload
%autoreload 2
import models
inv_yhat, inv_y, rmse=models.make_lstm_prediction(test_X,test_y,model,scaler)
print('LSTM Moddel on Test Data RMSE: %.3f' % rmse)
The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
LSTM Moddel on Test Data RMSE: 9.211